Hash suppression

ABSTRACT

An example method is provided in according with one implementation of the present disclosure. The method comprises generating, via a processor, a set of hashes for each of a plurality of objects. The method also comprises computing, via the processor, a high-dimensional sparse vector for each object, where the vector represents the set of hashes for each object. The method further comprises computing, via the processor, a combined high-dimensional sparse vector from the high-dimensional sparse vectors for all objects and computing a hash suppression threshold. The method also comprises determining, via the processor, a group of hashes to be suppressed by using the hash suppression threshold, and suppressing, via the processor, the group of selected hashes when performing an action.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims the priority benefitof, U.S. patent application Ser. No. 16/073,961 filed Jul. 30, 2018,which is 371 of PCT/US2015/065339 filed Dec. 11, 2015. The disclosure ofthe above-referenced application is incorporated herein in its entiretyfor all purposes.

A variety of analytic tasks may be performed on data (e.g., big datathat generally exceeds the processing capacity of conventional systems),and the results may be provided to a user. The analytics tasks mayinclude creating and running queries, indexing, retrieval, clustering,pattern detection, classification, and others.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an example system for suppressinglow value hashes and updating a hash index in accordance with animplementation of the present disclosure.

FIG. 2 illustrates a flowchart showing an example of a method forsuppressing low value hashes in accordance with an implementation of thepresent disclosure.

FIG. 3 illustrates a flowchart showing an example of a method fordetermining a hash suppression threshold in accordance with animplementation of the present disclosure,

FIG. 4 illustrates a flowchart showing an example of a method forevaluating and suppressing hashes of a new object in accordance with animplementation of the present disclosure.

FIG. 5 is an example block diagram illustrating a computer-readablemedium in accordance with an implementation of the present disclosure.

DETAILED DESCRIPTION OF SPECIFIC EXAMPLES

Many entities (e.g., enterprises, organizations) utilize databases forstorage of data relating to the entities. For example, a business maymaintain a database of customer information, and the customerinformation may be accessed by querying the database. Further, entitiesmay generally store vast amounts of data originating from theirbusiness, including operations data, customer feedback data, financialdata, Human Resource data, and so forth. Data stored in these databasesmay be accessed and updated for various purposes.

As described above, data stored in a database may be accessed andanalyzed in real-time for various purposes. For example, methods foranalyzing big data are of great interest in many applications, includingsimilarity search, collaborative filtering, data compression, retrieval,clustering, and classification.

One technique for analyzing large amounts of data is hashing and, forexample, hash-based indexing. In general, hashing is the transformationof a data object (e.g., vectors of numbers, string of characters, etc.)into at least one shorter fixed-length value or key (i.e., a hash) thatrepresents the original data object. In many situations, a plurality ofhashes may be generated from a data object. As used herein, the terms“data object” and “object” are to be used interchangeably and refer to adata component (e.g., vector of numbers) that, for example, may bestored in a database. A specific example of a hash is a concomitant rankorder (“CRO”) hash. Hashing may be used to index and retrieve objects ina database because two data objects that are similar tend to have morehashes in common and, therefore, using hashes for indexing may help toretrieve objects by similarity.

However, when analysis is performed on very large amounts of data, theplurality of objects in the database(s) may generate multiple hashesmany of which are of low importance. For example, many of the hashes mayhave IDs of multiple objects assigned to them. When a hash has multipleobject IDs assigned to it that means that the hash was generated frommultiple data objects. In that situation, it may be difficult to usesuch hashes to differentiate between objects because many objects sharethe same hash. Such hashes may be identified as low importance hashesand may be desirable to remove or suppress these low important hashesbefore performing any analytic action with the different types of data.

As noted above, hash-based indexing has proved to be a valuabletechnique for a variety of analytical operations includingsimilarity-based search and clustering. In all these applications, it isbeneficial to reduce the size of the hash index and improve thediscriminatory power of the hashes by eliminating the hashes that havelow importance and are associated with an excessive number of objects.By suppressing these low importance hashes, the size of the hash indexmay be significantly reduced and this will also improve the retrievalaccuracy of all analytic processes. As an example, for face retrievalanalysis hash suppression enhances the efficiency and accuracy of theretrieval algorithm.

Existing tools and techniques for indexing high velocity data streams inreal time are not adequate to efficiently update an incremental hashindex and improve the performance and the accuracy of the data analysis.In this regard, according to examples, computer-implemented system andtechniques for suppressing low value hashes and incrementally updating ahash index are disclosure herein. In one example, the proposedtechniques may suppress low-discriminative hashes in a high velocitydata stream before performing an action with the objects in the datastream (e.g., classification, clustering, etc.).

In one implementation, the techniques described herein proposesuppressing selected hashes for a group of data objects (e.g., CROhashes) and, therefore, pruning the hash set generated for each givenobject. The proposed techniques enable updating and maintaining anincremental hash index with high discriminative power resulting inbetter analysis of the data in terms of accuracy and efficiency forvarious tasks such as retrieval, classification, clustering, etc.Therefore, using the proposed techniques improves the performance of thetask that follows the hashing process.

In one example, a processor may generate a set of hashes for each of aplurality of objects and may compute a high-dimensional sparse vectorfor each object, where the vector represents the set of hashes for eachobject. The processor may also compute a combined high-dimensionalsparse vector from the high-dimensional sparse vectors for all objectsand may compute a hash suppression threshold. The processor may furtherdetermine a group of hashes to be suppressed by using the hashsuppression threshold and may suppress the group of selected hashes whenperforming a task or an action (e.g., clustering, classification, etc.).

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof, and in which is shownby way of illustration specific examples in which the disclosed subjectmatter may be practiced. It is to be understood that other examples maybe utilized and structural or logical changes may be made withoutdeparting from the scope of the present disclosure. The followingdetailed description, therefore, is not to be taken in a limiting sense,and the scope of the present disclosure is defined by the appendedclaims. Also, it is to be understood that the phraseology andterminology used herein is for the purpose of description and should notbe regarded as limiting. The use of “including,” “comprising” or“having” and variations thereof herein is meant to encompass the itemslisted thereafter and equivalents thereof as well as additional items.Furthermore, the term “based on, as used herein, means based at least inpart on.” It should also be noted that a plurality of hardware andsoftware based devices, as well as a plurality of different structuralcomponents may be used to implement the disclosed methods and devices.

Referring now to the figures, FIG. 1 is a schematic illustration of anexample system 10 for suppressing low value hashes and updating a hashindex. The illustrated system 10 is capable of carrying out thetechniques described below. As shown in FIG. 1 , the system 10 isdepicted as including at least one computing device 100 (e.g.,application server, compute node, desktop or laptop computer, smartphone, etc.). In the embodiment of FIG. 1 , computing device 100includes a processor 102, an interface 106, and a machine-readablestorage medium 110. Although only computing device 100 is described indetails below, the techniques described herein may be performed byseveral computing devices or by engines distributed on differentdevices. Thus, the computing device 100 may or may not be an independentcomputing device. The computing device 100 may include additionalcomponents and some of the components depicted therein may be removedand/or modified without departing from a scope of the system that allowsfor carrying out the functionality described herein.

In one example, the computing device 100 (or another computing device)may communicate with a data corpus 150 and with an interactive userinterface 160 (e.g., graphical user interface). The data corpus 150 mayinclude different types of data objects. The data in the data corpus 150may include text-like data, categorical data, numerical data, structureddata, unstructured data, or any other type of data. The device 100 mayreceive an incoming data stream of data objects (e.g., vectors ofnumbers, etc.) from the data corpus 150.

The computing device 100 may implement engines 120-140 (and componentsthereof) in various ways, for example as hardware and programming. Eachof the engines 120-140 may include, for example, a hardware deviceincluding electronic circuitry for implementing the functionalitydescribed below, such as control logic and/or memory. In addition or asan alternative, the engines 120-140 may be implemented as anycombination of hardware and software to implement the functionalities ofthe engines. The programming for the engines 120-140 may take the formof processor-executable instructions stored on a non-transitorymachine-readable storage medium and the hardware for the engines 120-140may include a processing resource to execute those instructions. Aprocessing resource may include a number of processors and may beimplemented through a single processor or multi-processor architecture.In an alternative example, engines 120-140 may be distributed betweenthe computing device 100 and other computing devices. It is to beunderstood that the operations described as being performed by theengines 120-140 of the computing device 100 that are related to thisdescription may, in some implementations, be performed by externalengines (not shown) or distributed between the engines of the computingdevice 100 and other electronic/computing devices.

Processor 102 may be central processing unit(s) (CPUs),microprocessor(s), and/or other hardware device(s) suitable forretrieval and execution of instructions (not shown) stored inmachine-readable storage medium 110. Processor 102 may fetch, decode,and execute instructions to identify different groups in a dataset. Asan alternative or in addition to retrieving and executing instructions,processor 102 may include electronic circuits comprising a number ofelectronic components for performing the functionality of instructions.

Interface 106 may include a number of electronic components forcommunicating with various devices. For example, interface 106 may be anEthernet interface, a Universal Serial Bus (USB) interface, an IEEE 1394(Firewire) interface, an external Serial Advanced Technology Attachment(eSATA) interface, or any other physical connection interface suitablefor communication with the computing device. Alternatively, interface106 may be a wireless interface, such as a wireless local area network(WLAN) interface or a near-field communication (NFC) interface that isused to connect with other devices/systems and/or to a network. The userinterface 160 and the computing device 100 may be connected via anetwork. In one example, the network may be a mesh sensor network (notshown). The network may include any suitable type or configuration ofnetwork to allow for communication between the computing device 100, theuser interface 160, and any other devices/systems (e.g., other computingdevices, displays), for example, to send and receive data to and from acorresponding interface of another device.

In one example, the hash generating engine 120 may generate a set ofhashes for each of a plurality of objects (e.g., received from the datacorpus 150). As noted above, the data corpus may include various dataobjects. Various techniques may be used to generate the set of hashesfor each object. In one example, the hashes are CRO hashes.

The vector generating engine 130 may compute a high-dimensional sparsevector for each object, where the vector represents the set of hashesfor each object. In addition, the vector generating engine 130 maycompute a combined high-dimensional sparse vector from thehigh-dimensional sparse vectors for all objects, where the combinedhigh-dimensional sparse vector indicates how many individual objectsinclude each hash.

The hash suppression engine 140 may compute a hash suppressionthreshold, may determine a group of hashes to be suppressed by using thehash suppression threshold, and may suppress the group of selectedhashes when performing an action (e.g., clustering, etc.). For example,the hash suppression engine 140 may compute a sum and a sum of squaresof all elements of the combined high-dimensional sparse vector, maycompute a mean and a standard deviation of all elements of the combinedhigh-dimensional sparse vector, and may using the mean, the standarddeviation, and a hash suppression factor to compute the hash suppressionthreshold.

In addition, the hash suppression engine 140 may determine whether thehashes for a new object are included in the group of suppressed hashesand may remove the suppressed hashes from the set of hashes for the newobject. That way, the hash suppression threshold may be computedincrementally, the system may process each new objects as it comes in,and may update the group of suppressed hashes without analyzing thepreviously ingested objects.

FIG. 2 illustrates a flowchart showing an example of a method 200 forsuppressing low value hashes. Although execution of the method 200 isdescribed below with reference to the system 10, the components forexecuting the method 200 may be spread among multiple devices/systems.The method 200 may be implemented in the form of executable instructionsstored on a machine-readable storage medium, and/or in the form ofelectronic circuitry.

In one example, the method 200 can be executed by at least one processor(e.g., processor 102 of device 100). In other examples, the method maybe executed by another processor in communication with the system 10.Various elements or blocks described herein with respect to the method200 are capable of being executed simultaneously, in parallel, or in anorder that differs from the illustrated serial manner of execution. Themethod 200 is also capable of being executed using additional or fewerelements than are shown in the illustrated examples.

The method 200 begins at 210, where a processor may generate a set ofhashes (e.g., CRO hashes) for each of a plurality of objects. Varioustechniques may be used to generate the hashes for each object. Forexample, a processor may receive an input set of objects I={O₁, O₂ . . .O_(N)} from the data corpus 150. Then, the processor may generate a hashset h(O) for each object O_(i) from the set of objects I, where theelements of h(O) may be drawn from a large hash universe of numbers(e.g., 1 . . . U). In other words, for each object O_(i) the processormay generate a set of hashes h(O) or numbers that have values between1-U. In one example, when ten hashes are generated for each objectO_(i), then h(O) may be a set of ten numbers where each number is a hash(i.e., each number is between 1-U).

At 220, the processor may compute a high-dimensional sparse vector O*for each object O_(i), where the vector represents the set of hashesh(O) for each object O_(i). In other words, the processor may transformthe data object O_(i) to a high-dimensional sparse binary vector O*having a dimension U. For example, each element of the dimension in thevector may be a real number and the dimension U may be very large. Inone implementation, O* is an array or vector of length U, where allnumbers are zero except the places or locations corresponding with thehashes in the hash set h(O) for each object. So, if the hash set h(O)includes number 1, 2 and 5, then O* is a vector having various zerovalues except at locations 1, 2, and 5. In one implementation, thehigh-dimensional sparse binary vector O* may be computed by using thefollowing equation:

$\begin{matrix}{{O^{*}\lbrack i\rbrack} = \left\{ \begin{matrix}{{1{if}i} \in {h(O)}} \\{{0{if}i} \notin {h(O)}}\end{matrix} \right.} & \left( {{Eqn}.1} \right)\end{matrix}$

Next, the processor may compute a combined high-dimensional sparsevector from the high-dimensional sparse vectors O* for all objects (at230). For example, the processor may add all the vectors for all theobject at each of their coordinates/locations. The result is a combinedhigh-dimensional sparse vector, where each of the coordinates orelements of the new vector is a sum of the coordinates of alladded/existing vectors. In one implementation, the combinedhigh-dimensional sparse vector may be computed by using the followingequation:C=Σ _(i=1) ^(N) O* _(i)  (Eqn. 2)

In Equation 2, C represents the combined high-dimensional sparse vectorand N represents the number of input objects (e.g., vectors of numbers).In one example, the combined high-dimensional sparse vector indicateshow many individual objects include each hash. In other words, thecombined high-dimensional sparse vector may show how many times each ofthe hashes is used or included in each of the plurality of data objects.

At 240, the processor may compute a hash suppression threshold. Asexplained in additional details below, the hash suppression thresholdmay be used to identify a set of hashes to be suppressed by theprocessor. An example technique for computing the hash suppressionthreshold is described below in relation to FIG. 3 .

At 250, the processor may determine a group of hashes to be suppressedby using the hash suppression threshold. In one example, the processormay determine a group of hashes to be suppressed by applying thefollowing equation:B={i:C[i]>τ}  (Eqn. 3)

In equation 3, B represents the group of hashes to be suppressed (i.e.,the set of coordinates for the combined high-dimensional sparse vectorC), and τ is the hash suppression threshold. For example, when thehashes for the objects are generated, the processor may compare allhashes to the hash suppression threshold. If it is determined that ahash is above the hash suppression threshold, the processor may flagthat hash for removal/suppression. In other words, the processorcomputes a list of hashes that should be suppressed, i.e. they shouldnot be used for indexing objects and should be ignored during the dataretrieval and analysis process.

Next, the processor may suppress the group of selected hashes whenperforming an action (e.g., retrieval, classification, etc.) (at 260).For example, any time the system attempts to perform an action (e.g.,based on a user request via the user interface 160), the processor maysuppress the group of selected hashes and then proceeds to perform theaction. In the alternative, the processor may consistently suppress thegroup of selected hashes in scheduled or unscheduled intervals withoutan actual request for an action. When a hashes for a new object aregenerated, the processor may check whether these hashes are in the groupof hashes to be suppressed and may remove them from the hash setassociated to the object if they are. As noted above, the benefit ofthis is that the hashes that have a large number of associated objectsare not very useful as discriminators between objects. By suppressingthese hashes, the processor significantly reduces the size of the hashindex, and also improves the retrieval/analysis accuracy of the data.

In one example, the process described above may be executed as a batchalgorithm (i.e., where all the objects in the set are known beforehand).In another example, the techniques described above may also be appliedto streaming applications, where data objects are received in asequential manner. In that example implementation, an incrementalversion of the algorithm would process each new data object as it comesin, and will update the threshold and the group of selected hasheswithout having to touch the previously evaluated objects. An example ofthis incremental update is described in more details below in relationto FIG. 5 .

FIG. 3 illustrate a flowchart showing an example of a method 300 fordetermining a hash suppression threshold. Although execution of themethod 300 is described below with reference to the system 10, thecomponents for executing the method 300 may be spread among multipledevices/systems. The method 300 may be implemented in the form ofexecutable instructions stored on a machine-readable storage medium,and/or in the form of electronic circuitry. In one example, the method300 can be executed by at least one processor of a computing device(e.g., processor 102 of device 100).

The method 300 begins at 320, where a processor may compute a sum and asum of squares of all elements (i.e., coordinates) of the combinedhigh-dimensional sparse vector C. In one example, the sum and a sum ofsquares of all elements of the combined high-dimensional sparse vectormay be computed by using the following equations:S=Σ _(k=1) ^(U) C[k]  (Eqn. 4)V=Σ _(k=1) ^(U) C ²[k]  (Eqn. 5)

In equations 4 and 5, S represents the sum of all elements in thecombined vector C and V represents the sum of squares of all elements inthe combined vector C. In some examples, the sum S of all elements inthe combined vector C and the sum of squares of all elements may be areal number (e.g., the total number of hashes in all the objects).

Next, the processor may compute a mean and a standard deviation of allelements of the combined high-dimensional sparse vector (at 330). In oneexample, the mean and the standard deviation of all elements of thecombined high-dimensional sparse vector may be computed by using thefollowing equations:

$\begin{matrix}{\mu = \frac{S}{U}} & \left( {{Eqn}.6} \right)\end{matrix}$ $\begin{matrix}{\sigma = \sqrt{\frac{V}{U} - \mu^{2}}} & \left( {{Eqn}.7} \right)\end{matrix}$

In equations 6 and 7, μ represents the mean of all elements of thecombined high-dimensional sparse vector, σ represents the standarddeviation of all elements of the combined high-dimensional sparsevector, and U represents the total number of possible hashes. In oneexample, the mean μ represents the average count that every hash isfound in X number of objects and the standard deviation σ represents howthe hashes are dispersed between the plurality of objects.

At 340, the processor may use the mean, the standard deviation, and ahash suppression factor to compute the hash suppression threshold. Inone example, the hash suppression threshold may be computed by using thefollowing equation:τ=μ+ασ  (Eqn. 8)

In equation 8, μ represents the hash suppression threshold. As shown,the processor uses the mean and the standard deviation of all elementsof the combined high-dimensional sparse vector, plus a hash suppressionfactor to compute the hash suppression threshold. The hash suppressionfactor α may be a constant number (e.g., 1.5, etc.) used to assist withthe computation of μ. As noted above, by suppressing low value hashesthe processor not only increases the discriminative power of hashingalgorithm, but it also reduces the computational cost of differentactions (e.g., retrieval, clustering, classification, etc.) that followthe hashing.

FIG. 4 illustrates a flowchart showing an example of a method 400 forevaluating and suppressing hashes of a new object. Although execution ofthe method 400 is described below with reference to the system 10, thecomponents for executing the method 400 may be spread among multipledevices/systems. The method 400 may be implemented in the form ofexecutable instructions stored on a machine-readable storage medium,and/or in the form of electronic circuitry. In one example, the method400 can be executed by at least one processor of a computing device(e.g., processor 102 of device 100).

The method 400 begins at 410, where the processor may generate a set ofhashes for a new object. This process may be similar to the processdescribed in block 210 of method 200. The processor may determinewhether the hashes for the new object are included in the group ofsuppressed hashes (at 420). In one example, this is the group ofsuppressed hashes that was previously identified by the processor. Ifthe hashes for the new object are not included in the group ofsuppressed hashes, the processor may stop (at 430). If, however, thehashes for the new object are included in the group of suppressedhashes, the processor may remove the identified suppressed hashes fromthe set of hashes for the new object (at 440).

Next, the processor may update the combined high-dimensional sparsevector to include the high-dimensional sparse vector for the new object(at 450). An updated combined high-dimensional sparse vector may becomputed by using techniques similar to the techniques described inblock 230 of the method 200. At 460, the processor may compute anupdated hash suppression threshold (e.g., by using the new update thecombined high-dimensional sparse vector). The updated hash suppressionthreshold may be computed by using techniques similar to the techniquesdescribed in block 240 of the method 200. At 470, the processor maydetermine an updated group of hashes to be suppressed by using theupdated hash suppression threshold (e.g., similarly to the processdescribed in FIG. 3 ).

FIG. 5 illustrates a computer 501 and a non-transitory machine-readablestorage medium 505 according to an example. In one example, the computer501 maybe similar to the computing device 100 of the system 10 or mayinclude a plurality of computers. For example, the computer may be aserver computer, a workstation computer, a desktop computer, a laptop, amobile device, or the like, and may be part of a distributed system. Thecomputer may include one or more processors and one or moremachine-readable storage media. In one example, the computer may includea user interface (e.g., touch interface, mouse, keyboard, or gestureinput device).

Computer 501 may perform methods 200-400 and variations thereof.Additionally, the functionality implemented by computer 501 may be partof a larger software platform, system, application, or the like.Computer 501 may be connected to a database (not shown) via a network.The network may be any type of communications network, including, butnot limited to, wire-based networks (e.g., cable), wireless networks(e.g., cellular, satellite), cellular telecommunications network(s), andIP-based telecommunications network(s) (e.g., Voice over InternetProtocol networks). The network may also include traditional landline ora public switched telephone network (PSTN), or combinations of theforegoing.

The computer 501 may include a processor 503 and non-transitorymachine-readable storage medium 505. The processor 503 (e.g., a centralprocessing unit, a group of distributed processors, a microprocessor, amicrocontroller, an application-specific integrated circuit (ASIC), agraphics processor, a multiprocessor, a virtual processor, a cloudprocessing system, or another suitable controller or programmabledevice) and the storage medium 505 may be operatively coupled to a bus.Processor 503 can include single or multiple cores on a chip, multiplecores across multiple chips, multiple cores across multiple devices, orcombinations thereof.

The storage medium 505 may include any suitable type, number, andconfiguration of volatile or non-volatile machine-readable storage mediato store instructions and data. Examples of machine-readable storagemedia include read-only memory (“ROM”), random access memory (“RAM”)(e.g., dynamic RAM [“DRAM”], synchronous DRAM [“SDRAM”]), electricallyerasable programmable read-only memory (“EEPROM”), magnetoresistiverandom access memory (MRAM), memristor, flash memory, SD card, floppydisk, compact disc read only memory (CD-ROM), digital video disc readonly memory (DVD-ROM), and other suitable magnetic, optical, physical,or electronic memory on which software may be stored.

Software stored on the non-transitory machine-readable storage medium505 and executed by the processor 503 includes, for example, firmware,applications, program data, filters, rules, program modules, and otherexecutable instructions. The processor 503 retrieves from themachine-readable storage medium 505 and executes, among other things,instructions related to the control processes and methods describedherein.

The processor 503 may fetch, decode, and execute instructions 507-511among others, to implement various processing. As an alternative or inaddition to retrieving and executing instructions, processor 503 mayinclude at least one integrated circuit (IC), other control logic, otherelectronic circuits, or combinations thereof that include a number ofelectronic components for performing the functionality of instructions507-511. Accordingly, processor 503 may be implemented across multipleprocessing units and instructions 507-511 may be implemented bydifferent processing units in different areas of computer 501.

The instructions 507-511 when executed by processor 503 (e.g., via oneprocessing element or multiple processing elements of the processor) cancause processor 503 to perform processes, for example, methods 200-400,and/or variations and portions thereof. In other examples, the executionof these and other methods may be distributed between the processor 503and other processors in communication with the processor 503. In oneimplementation, instructions 507-511 may represent incrementaltechniques that process each new data object as it comes in, and updatethe threshold and the group of selected hashes without having to touchthe previously evaluated objects.

For example, hash generating instructions 607 may cause processor 503 togenerate a set of hashes for a first object. These instructions mayfunction similarly to the techniques described in block 210 of method200.

Vector generating instructions 509 may cause the processor 503 tocompute a high-dimensional sparse vector for the first object, where thevector represents the set of hashes for each object. These instructionsmay function similarly to the techniques described block 220 of method200. Further, vector generating instructions 509 may cause the processor503 to compute a combined high-dimensional sparse vector. The combinedhigh-dimensional sparse vector indicates how many individual objectsinclude each hash. These instructions may function similarly to thetechniques described block 230 of method 200.

Hash suppression instructions may cause the processor 503 to compute ahash suppression threshold and to suppress the group of selected hashesthat are above the hash suppression threshold when performing an action.These instructions may function similarly to the techniques describedblocks 240-260 of method 200.

When a new object arrives, the hash generating instructions 607 maycause processor 503 to generate a set of hashes for the new/secondobject. Vector generating instructions 509 may cause the processor 503to compute a high-dimensional sparse vector for the second object and toupdate the combined high-dimensional sparse vector to include the highdimensional sparse vector for the second object. These instructions mayfunction similarly to the techniques described block 220 of method 200and the techniques described block 450 of method 400.

The hash suppression instructions 511 may also cause the processor 503to compute an updated hash suppression threshold and to determine anupdated group of hashes to be suppressed by using the updated hashsuppression threshold. These instructions may function similarly to thetechniques described blocks 450-470 of method 400. That way, theinstructions of the computer 501 may incrementally update thesuppression threshold and the group of hashes to be suppressed for everynew object that arrives.

Hash suppression instructions 511 may further cause the processor 503 todetermine whether the hashes for the second object are included in theinitial the group of suppressed hashes, and to remove the suppressedhashes from the set of hashes for the second object. These instructionsmay function similarly to the techniques described blocks 410-440 ofmethod 400.

In the foregoing description, numerous details are set forth to providean understanding of the subject matter disclosed herein. However,implementations may be practiced without some or all of these details.Other implementations may include modifications and variations from thedetails discussed above. It is intended that the appended claims coversuch modifications and variations.

What is claimed is:
 1. A method comprising: generating, via a processor,a hash for an object; computing, via the processor, a vector for theobject, wherein the vector represents the hash for the object;computing, via the processor, a combined vector from the vector andadditional vectors of one or more different objects; computing, via theprocessor, a hash suppression threshold; determining, via the processor,whether the hash should be suppressed by using the hash suppressionthreshold; suppressing, via the processor, the hash when performing anaction; generating, via the processor, a hash for a new object;determining, via the processor, whether the hash for the new object isincluded in the group of suppressed hashes; removing, via the processor,the suppressed hash if the hash is included in the group of suppressedhashes; updating, via the processor, the combined vector to include thevector for the new object; computing, via the processor, an updated hashsuppression threshold; and determining, via the processor, a hash to besuppressed by using the updated hash suppression threshold.
 2. Themethod of claim 1, wherein the combined vector indicates how manyobjects include each hash.
 3. The method of claim 1, wherein computingthe hash suppression threshold comprises: computing, via the processor,a sum and a sum of squares of all elements of the combined vector;computing, via the processor, a mean and a standard deviation of allelements of the combined vector; using the mean, the standard deviation,and a hash suppression factor to compute the hash suppression threshold.4. The method of claim 1, wherein the vectors are high-dimensionalsparse vectors.
 5. A system, comprising: a processor; and a memoryincluding instructions that, when executed by the processor, cause thesystem to: generate a set of hashes for one or more data objects;compute a vector for each of the one or more data objects, wherein thevector represents the set of hashes for each of the one or more dataobjects, compute a combined vector from the vectors for all of the oneor more data objects, wherein the combined vector indicates how manyindividual data objects include each hash; incrementally compute a hashsuppression threshold, determine a group of hashes to be suppressed byusing the hash suppression threshold, suppress the group of selectedhashes when performing an action, generate a new incremented set ofhashes for each object; determine whether the new incremented set ofhashes for each of the one or more data objects are included in thegroup of suppressed hashes; remove the suppressed hashes from the newincremented set of hashes for each of the one or more data objects;update the combined vector to include the vector for each of the one ormore data objects; compute an updated hash suppression threshold by eachincrement; and determine an updated group of hashes to be suppressed byusing the updated hash suppression threshold.
 6. The system of claim 5,wherein the hash suppression engine is further to: compute a sum and asum of squares of all elements of the combined vector; compute a meanand a standard deviation of all elements of the combined vector; and usethe mean, the standard deviation, and a hash suppression factor tocompute the hash suppression threshold.
 7. The system of claim 5,wherein vectors are high-dimensional sparse vectors.
 8. The system ofclaim 5, wherein the data objects are executed as a batch algorithm,such that all data objects are known before execution.
 9. The system ofclaim 5, wherein the data objects are executed as streamingapplications, such that data objects are received in a sequentialmanner.
 10. The system of claim 5, wherein the system would process eachdata object incrementally, and update the threshold and the group ofselected hashes without changing the previously evaluated objects.
 11. Anon-transitory machine-readable storage medium encoded with instructionsexecutable by at least one processor, the machine-readable storagemedium comprising instructions to: generate a set of hashes for a firstobject; compute a high-dimensional sparse vector for the first object,wherein the vector represents the set of hashes for each object; computea combined high-dimensional sparse vector; compute a hash suppressionthreshold; determine a group of hashes to be suppressed by using thehash suppression threshold; consistently suppress the group of selectedhashes that are above the hash suppression threshold when performing anaction; generate a new set of hashes for a second object and compute ahigh-dimensional sparse vector for the second object; determine whetherthe new set of hashes for each of the one or more data objects areincluded in the group of suppressed hashes; remove the suppressed hashesfrom the new set of hashes for each of the one or more data objects;update the combined high-dimensional sparse vector to include the highdimensional sparse vector for the second object; compute an updated hashsuppression threshold; and determine an updated group of hashes to beconsistently suppressed by using the updated hash suppression threshold.12. The non-transitory machine-readable storage medium of claim 11,further comprising instructions to: determine whether the hashes for thesecond object are included in the initial group of suppressed hashes;and remove the suppressed hashes from the set of hashes for the secondobject.
 13. The non-transitory machine-readable storage medium of claim11, wherein the combined high-dimensional sparse vector indicates howmany individual objects include each hash.
 14. The non-transitorymachine-readable storage medium of claim 11, further comprisinginstructions to: compute a sum and a sum of squares of all elements ofthe combined high-dimensional sparse vector; compute a mean and astandard deviation of all elements of the combined high-dimensionalsparse vector; and use the mean, the standard deviation, and a hashsuppression factor to compute the hash suppression threshold.
 15. Thenon-transitory machine-readable medium of claim 11, wherein consistentlysuppressing the group of selected hashes occurs at scheduled intervals.16. The non-transitory machine-readable medium of claim 11, whereinconsistently suppressing the group of selected hashes occurs atunscheduled intervals.